Real-Time Active Human Tracking by Hierarchical Integration of Audition and Vision

We present a real-time multiple human tracking system for a humanoid with a pair of microphones and a pair of cameras. The key idea for robust recognition and scalability of multi-modality is the hierarchical integration of multiple sensory information. First, sound direction by auditory localization module, speaker ID by spear identi cation module, face location, face ID, and object location by stereo vision, and motor direction are extracted as separate events. Next, each stream is formed as a temporal sequence of events. Then, di erent streams are associated according to their proximity. This association process consists of two stages. At the rst stage, an association in location or name is performed. At the second stage, the higher level association between locationand name-associated streams may be performed. The hierarchical association enables integration of various kinds of streams and resolve ambiguities of lower level perceptions. The focus-of-attention is mainly controled by hierarchical association streams. As a result, the humanoid demonstrates robust human tracking even when several persons speak simultaneously.